This document describes how to quickly get started with the precision debugging tool MindStudio Probe (msProbe) during the training development process in the PyTorch scenario.
For foundation models developed based on Ascend or migrated from GPU to Ascend NPU, training issues such as precision overflow/underflow, loss divergence, or non-convergent loss may arise. Since metrics such as the training loss cannot accurately locate the failed module, msProbe is recommended for rapid fault demarcation.
When using msProbe for model precision debugging, perform the following operations:
Configuration check before training
Identify the configuration differences that affect the precision in two environments.
Training status monitoring
Monitor exceptions that occur during computing, communication, and optimizer operations during training.
Precision data collection
Collect the forward and backward input and output data at the API or module level during training.
Precision pre-check
Scan API data to identify APIs with precision issues.
Precision comparison
Compare the API data on NPU with that in the benchmark environment to quickly locate precision issues.
This Quick Start guide focuses on rapid onboarding for precision data collection and precision comparison. For usage details of other tool functions, please refer to the relevant documentation.
Prepare a training server equipped with Ascend NPUs (such as Atlas A2 training servers) and install the NPU driver and firmware.
Install the CANN Toolkit and OPS (operator package) of the matching version and configure CANN environment variables. The following uses CANN 8.5.0 as an example. For details, see .
Install the framework.
In the following example, PyTorch 2.9.0, Python 3.12, AArch64-based system, and torchvision 0.24.0 are used as examples in the PyTorch training scenario. For details, see "Installing PyTorch > " in .
Install msProbe by referring to .
[object Object]
In this example, the ResNet-50 model and virtual data are used for training.
Prerequisites
Data Collection
Prepare a training script.
[object Object]is used as an example. In the GPU and Ascend NPU environments, you can directly copy the sample code from . When training is performed in the GPU environment, add the following lines 24 and 25 to the script.[object Object]Create a configuration file.
Create a
[object Object]file in the directory where the training script is located. Copy the file content as follows:[object Object]Add the tool to the training script (
[object Object]) in the GPU and Ascend NPU environments.[object Object]
[object Object][object Object]
Run the training script command. The tool collects precision data during model training.
[object Object]If the following information is displayed in the log, you can manually stop model training and view the collected data to save time.
[object Object]
Checking the Result
The following directory structure is displayed in the path specified by [object Object]. Select data for analysis as required.
The collected data needs to be further analyzed for precision comparison. For details, see .
Prerequisites
- Complete procedures listed in .
- Complete procedures listed in to obtain the precision data in the GPU and Ascend NPU environments.
Performing Comparison
Prepare data.
After dumping data in the GPU and Ascend NPU environments, copy the precision data dumped from the GPU environment to the Ascend NPU environment. Pay attention to the directory names specified by
[object Object].[object Object]and[object Object]are used as examples.The path of
[object Object]in the[object Object]directory is[object Object].The path of
[object Object]file in the[object Object]directory is[object Object].Perform the comparison.
The command is as follows:
[object Object]If the following information is displayed, the comparison is successful:
[object Object]Analyze the comparison result file.
[object Object]generates the following file in[object Object]:[object Object]: This file lists the details about all APIs for precision comparison and comparison results. You can locate suspicious operators based on the comparison result ([object Object]) and error message ([object Object]). However, each metric has a determination standard. Since each metric has its own evaluation criteria, make judgments based on actual circumstances.Examples:
Figure 1 compare_result_1
Figure 2 compare_result_2
Figure 3 compare_result_3
For more information about the comparison result analysis, see .
Prerequisites
Complete procedures listed in to obtain the precision data in the GPU and Ascend NPU environments.
For hierarchical visualization, the
[object Object]parameter in the[object Object]file must be set to[object Object]or[object Object]during a data dump. In this example,[object Object]is used to re-collect precision data.
Performing Comparison
Prepare data.
After dumping data in the GPU and Ascend NPU environments, copy the precision data dumped from the GPU environment to the Ascend NPU environment. Pay attention to the directory names specified by
[object Object].[object Object]and[object Object]are used as examples.The path of
[object Object]is[object Object].The path of
[object Object]is[object Object].Perform graph comparison.
[object Object]After the comparison is complete, a
[object Object]file is generated in the[object Object]directory.Start TensorBoard.
[object Object]--The path specified by
[object Object]is[object Object]in step 2.After the preceding command is executed, the following log is displayed:
[object Object]Open a browser in the Windows environment and access
[object Object], where[object Object]should be replaced with the IP address of your server, for example,[object Object].After the access is successful, the TensorBoard page is displayed, as shown in the following figure.
Figure 1 Graph comparison in hierarchical visualization mode